Detection of change in posture in video

ABSTRACT

Input video data is processed to detect a change in a posture of a person shown in the video data. The change of posture may be the result of an event, for example, the person falling or getting up. The input video data may include a plurality of frames. Objects in the frames are tracked and then classified, for example, as human and non-human targets. At least one of the position or location of a human target in the frames is identified. Changes in the location or position of the human target between the frames is determined. When the change in at least of the position or location exceeds a predetermined threshold, a falling down event or a getting up event is detected. The changes in the position or location of the human target can be determined based on a number of different factors.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention generally relates to surveillance systems. Specifically,the invention relates to a video-based surveillance system that can beused, for example, to detect when a person falls or gets up.

2. Related Art

Some state-of-the-art intelligent video surveillance (IVS) system canperform content analysis on frames generated by surveillance cameras.Based on user-defined rules or policies, IVS systems may be able toautomatically detect events of interest and potential threats bydetecting, tracking and classifying the targets in the scene. For mostIVS applications the overall tracking of objects is sufficient: thatalready enables e.g. detecting when an object enters a restricted area,or when an object is left behind or taken away. In other applications,however, some further granularity is needed. The detection of change inposture, for example, when a person falls or gets up, is an example ofthis. Detecting such events is important in a wide range ofapplications, including dispatching help quickly, especially inhospitals, nursing homes, or in the homes of the sick or the elderly;for liability reduction; or in security applications when guardingpeople.

SUMMARY OF THE INVENTION

Embodiments of the invention include a method, a system, an apparatus,and an article of manufacture for automatic detection of change inposture. Such embodiments may involve computer vision techniques toautomatically detect the change of posture and other such events bydetecting, tracking, and analyzing people. This technology hasapplications in a wide range of scenarios.

Embodiments of the invention may include a machine-accessible mediumcontaining software code that, when read by a computer, causes thecomputer to perform a method for automatic detection of change inposture comprising the steps of: performing change detection on theinput surveillance video; detecting targets; tracking targets;classifying targets as human or non-human; optionally detecting andtracking the head of the tracked person; optionally tracking the bodyparts of the person; and detecting a change in posture.

A system used in embodiments of the invention may include a computersystem including a computer-readable medium having software to operate acomputer in accordance with embodiments of the invention.

An apparatus according to embodiments of the invention may include acomputer including a computer-readable medium having software to operatethe computer in accordance with embodiments of the invention.

An article of manufacture according to embodiments of the invention mayinclude a computer-readable medium having software to operate a computerin accordance with embodiments of the invention.

Exemplary features of various embodiments of the invention, as well asthe structure and operation of various embodiments of the invention, aredescribed in detail below with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features of various embodiments of the inventionwill be apparent from the following, more particular description of suchembodiments of the invention, as illustrated in the accompanyingdrawings, wherein like reference numbers generally indicate identical,functionally similar, and/or structurally similar elements.

FIGS. 1A and 1B depict how falling sideways may result in a significantchange in aspect ratio from a largely vertical standing position to alargely horizontal fallen position.

FIGS. 2 and 2B depict how falling towards or away from the camera mayresult in a significant change in the detected height of the trackedperson.

FIGS. 3A and 3B depict how the location of the topmost and bottommostpoint of the person may be used to eliminate false alarms caused by anocclusion.

FIGS. 4A and 4B depict how a person just bending down may result in afalse alarm.

FIGS. 5A and 5B depict how the trajectory of a tracked person may beused to eliminate missing the detection of a person falling towards thecamera.

FIGS. 6A-6D depict simple shape models the IVS system may use todescribe tracked objects.

FIGS. 7A-7C depict how the head location and position information mayhelp detecting change of posture events.

FIG. 8 illustrates a plan view of the video surveillance system of theinvention.

DEFINITIONS

A “video” refers to motion pictures represented in analog and/or digitalform. Examples of video include: television, movies, image sequencesfrom a video camera or other observer, and computer-generated imagesequences.

A “frame” refers to a particular image or other discrete unit within avideo.

An “object” refers to an item of interest in a video. Examples of anobject include: a person, a vehicle, an animal, and a physical subject.

An “activity” refers to one or more actions and/or one or morecomposites of actions of one or more objects. Examples of an activityinclude: entering; exiting; stopping; falling; getting up; moving;raising; lowering; growing; and shrinking.

A “place” refers to a space where an activity may occur. A location canbe, for example, scene-based or image-based. Examples of a scene-basedlocation include: a public space; a store; a retail space; an office; awarehouse; a hotel room; a hotel lobby; a lobby of a building; a casino;a bus station; a train station; an airport; a port; a bus; a train; anairplane; and a ship. Examples of an image-based location include: avideo image; a line in a video image; an area in a video image; arectangular section of a video image; and a polygonal section of a videoimage.

An “event” refers to one or more objects engaged in an activity. Theevent may be referenced with respect to a location and/or a time.

A “computer” refers to any apparatus that is capable of accepting astructured input, processing the structured input according toprescribed rules, and producing results of the processing as output.Examples of a computer include: a computer; a general purpose computer;a supercomputer; a mainframe; a super mini-computer; a mini-computer; aworkstation; a micro-computer; a server; an interactive television; ahybrid combination of a computer and an interactive television; andapplication-specific hardware to emulate a computer and/or software. Acomputer can have a single processor or multiple processors, which canoperate in parallel and/or not in parallel. A computer also refers totwo or more computers connected together via a network for transmittingor receiving information between the computers. An example of such acomputer includes a distributed computer system for processinginformation via computers linked by a network.

A “computer-readable medium” refers to any storage device used forstoring data accessible by a computer. Examples of a computer-readablemedium include: a magnetic hard disk; a floppy disk; an optical disk,such as a CD-ROM and a DVD; a magnetic tape; a memory chip; and acarrier wave used to carry computer-readable electronic data, such asthose used in transmitting and receiving e-mail or in accessing anetwork.

“Software” refers to prescribed rules to operate a computer. Examples ofsoftware include: software; code segments; instructions; computerprograms; and programmed logic.

A “computer system” refers to a system having a computer, where thecomputer comprises a computer-readable medium embodying software tooperate the computer.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

Exemplary embodiments of the invention are discussed in detail below.While specific exemplary embodiments are discussed, it should beunderstood that this is done for illustration purposes only. A personskilled in the relevant art will recognize that other components andconfigurations can be used without parting from the spirit and scope ofthe invention.

Detecting the change of posture of a person has several intelligentvideo surveillance (IVS) applications. Automatic, real-time detectionand alerting in case of a person falling or getting up can beadvantageous in a wide variety of situations. Such detection andalerting may enable dispatching help quickly and automatically in placeslike malls, stores, parking lots, assisted living communities,hospitals, or during duress monitoring in law enforcement. An alert mayeven be a life saver in the homes of elderly or sick people livingalone, who may not be able to get up and ask for help after falling. Insome of these scenarios the quick detection of the person slipping andfalling, and the ensuing fast response may help reduce liability aswell. Another liability aspect is to help avoid frivolous lawsuits bycustomers claiming to have fallen and having video evidence proving thecontrary or that the fall was intentional. The detection of a persongetting up also has several applications. Detection of a person gettingup can help guards in law enforcement. It can also be very helpful inhospitals, e.g. in intensive care units to detect if a person gets up,which if unnoticed can put the patient's life at risk.

The detection of falling and getting up are complementary problems. If avideo stream contains a falling event when played forward, the samevideo played backward will contain a getting up event. Hence the samealgorithmic considerations can be used for both cases. In thediscussions below, the focus is on describing the detection of falling,but a person skilled in the art can easily apply the same concepts andalgorithms to the detection of a person getting up.

In an exemplary embodiment of the invention, input video data isprocessed to detect a change in a posture of a person shown in the videodata. The change of posture may be the result of an event, for example,the person falling or getting up. The input video data may be of a placeand may include a plurality of frames. Objects in the frames are trackedand then classified, for example, as human and non-human targets. Atleast one of the position or location of a human target in the frames isidentified. Changes in the location or position of the human targetbetween the frames is determined. When the change in at least of theposition or location exceeds a predetermined threshold, a falling downevent or a getting up event is detected. The changes in the position orlocation of the human target can be determined based on a number ofdifferent factors, as discussed below.

An exemplary embodiment of the invention detects a person falling (orgetting up) based on a change in at least one of the height or aspectratio of the tracked person object. FIGS. 1A and 1B are examples offrames of video data of a place, for example a room in the home of anelderly person. As illustrated in FIGS. 1A and 1B, a person 101 standingis largely in a vertical position, while a person 102 who has fallendown is in largely horizontal position. The difference between thestanding and fallen positions can be detected using the aspect ratio,i.e. the ratio of the width (W) and the height (H) of the person. Asshown in FIG. 1A, the height of a standing person 101 is much greaterthan their width. Accordingly, the aspect ratio (W/H) of a typicalstanding person 101 is well below one. As shown in FIG. 1B, the heightof a fallen person is much less than their width. Accordingly, theaspect ratio of a person laying on the ground 102 is well above one.

The frames of the input video data may be processed to determine achange in the aspect ratio of a person. The height, location andposition of the person can be compared between the different frames.When the change in the aspect ratio exceeds a selected threshold, achange in posture, such as a falling down event if the change is anincrease in the aspect ratio, or a getting up event if the change is adecrease in the aspect ratio, may be detected. The amount of change inthe aspect ratio that indicates a change in posture depends on thespecific implementation. For example, the selected threshold may dependon the video camera parameters, such as the viewing angle. Indicating afalling event based on a relatively smaller change, for example, when1.2<larger aspect ratio/smaller aspect ratio <1.5, in the aspect ratiomay result in a greater number of false alarms compared to misseddetections. On the other hand indicating a falling event based on arelatively larger change may result in a greater number of misseddetections compared to false alarms. The threshold used may be based on,among other things, the place being monitored, the user's preferencesand their willingness to tolerate false alarms or missed detections.

FIG. 1B shows frame of video data in which a person has fallenessentially perpendicular to the camera viewing direction. If a personfalls substantially parallel to the camera viewing direction, that is,towards or away from the camera, as depicted in FIG. 2B, the change inaspect ratio might be less pronounced. The aspect ratio of the personmay be less than one both when a person is standing and when a personhas fallen. Basing the detection of a falling event only on a change inaspect ratio in such a case may result in an unacceptable number offalse alarms and/or missed detections. Thus, a change in the overallheight of the person object may be used to indicate that the person hasfallen.

As can be seen in FIG. 2A, the height 202 of the standing person 201 issignificantly greater than the height 204 of the fallen person 203. Forexample, a quotient of the greater height divided by the smaller heightthat is greater than 1.5 may indicate a “significantly greater” height.The frames of the video data can be processed to detect a change in theoverall height of the person. When the change in height exceeds apredetermined threshold, a change in posture is detected. The amount ofchange in the height that indicates a change in posture depends on thespecific implementation. The change in height can be used alone or inconjunction with a change in aspect ratio to detect a change in posture.A change in one or both of aspect ratio or height that exceedspredetermined thresholds may be required for an event to be detected.

False alarms and missed detections may occur, even if event detection isbased on a change in both aspect ratio and height. For example, asdepicted in FIGS. 3A and 3B, if an object occludes the bottom of theperson, the height of the person changes significantly (301 vs. 302),but a falling event has not occurred. Indicating a falling event in thiscase would be a false alarm. Some additional factors may help the IVSsystem to reduce false alarms and avoid missed detections. For examplethe locations of the top and bottom of the tracked object in the videodata can be used to avoid false alarms. In FIG. 3B, the person haswalked behind another object in the scene, for example a desk. Thebottommost point of the person (303 and 304) changes significantly whenthey pass behind the object, while the topmost point (305 and 306) doesnot change. This is in contrast to the falling down event, where asimilar change in height may occur, but is largely due to the top of theperson moving down with the bottommost point staying unchanged. Forexample, in the falling event shown in FIG. 2, the top of the personmoves towards the ground, with the bottom point remaining unchanged.Accordingly, a falling event may be indicated by the top of the objectmoving down, while the bottommost stays in place. The tracking of thetop and bottom of the object may be coupled with the requisite change inheight or aspect ratio to detect an event. For example, a change inheight that exceeds the predetermined threshold to indicate a fallingdown event may be measured. However, the top of the object does not movedown. This is the case shown in FIGS. 3A and 3B. The change in heightexceeds the predetermined threshold. However, a falling down event wouldnot be indicated in this case, as the top of the person did not movedown. Coupling the tracking of the top and bottom of the object with achange in height prevents a false alarm in this case.

In a further embodiment of the invention, the speed and/or the durationof the change in aspect ratio or height of the object may also helpavoid false alarms. FIG. 4 shows a standing person 401 and a personbending down 403. The height of a person may change significantly evenif the person only bends down, for example, to get an item from a bottomshelf in a store. The height 402 of the standing person 401 is clearlygreater than the height 404 of the person bending down 403. However,even though there is a change in the height, this is not a fallingevent. Falling is normally more sudden than bending down. Additionally,it typically takes a person longer to return to a standing positionafter falling down. The speed or duration of the change in aspect ratioor height should meet predefined limits in order for a falling event tobe indicated. The speed or duration of the change in the aspect ratio orheight that indicates a change in posture depends on the specificimplementation. The speed and/or the duration of the change in aspectratio or height of the object may be used alone or in any combinationwith the previously discussed factors to detect an event. For example,when a person bends down, a change in height occurs that indicates afalling event and the top of the person moves down, also indicating afalling event. However, the speed of the change does not meet thepredefined level. Therefore, a falling event is not indicated.

Another factor, a trajectory of the person, may be used to avoid amissed detection or false alarm. As depicted in FIGS. 5A and 5B, aperson falling towards a camera may be of approximately the same heightand aspect ratio whether standing (501) or on the ground (502), so theheight and aspect ratio based metrics may not detect a falling event inthis case. However, based on the trajectory of the person and trackingthe person through the frames of the video data, the system anticipatesthe tracked person to be in a particular location at a particular pointin time, such as at 503. However, at the time of the actual measurement,the person is not at location and position 503, but in a differentlocation and position, such as at 502. The sudden significantdiscrepancy, both the measured bottom (504) and top (505) being muchlower in the frame than what is expected (506 and 507), may indicate afalling event. Again, this factor may be used alone or in anycombination with the factors described above.

The above described methods provide the most natural approach if the IVSsystem tracks all targets with simple shape models, as depicted in FIGS.6A-D, e.g. using a bounding box (601), a centroid (602), a “footprint”(603), an ellipse (604), a convex hull (605), etc. This approach may beapplied to almost any previously stored forensic data (e.g. Lipton etal. Video Surveillance System, U.S. patent application Ser. Nos.09/987,707 and 11/057,154, which are incorporated herein by reference),since the data relied on (target location, width, height) is part theforensic storage.

If calibration information is available, that can further help thedetection. The calibration data basically tells the IVS system theexpected height of a standing person at any given location, so anydeviation from the calibration information may be an indication of achange in posture.

The robustness of the above described method(s) can be further improvedby incorporating head information into the decision making process. Inan exemplary embodiment the IVS system may detect and optionally trackthe head of the tracked person. Methods for the detection and trackingof the head of a person are known in the art. The location of the headand its position relative to the body may be used to detect the personfalling or getting up. As depicted in FIGS. 7A-C, the location of thehead 704 changes drastically between the standing (701) and fallen (702,703) postures. Unless falling away from the camera, the head positionrelative to the rest of the body also changes. In the standing positionthe head 704 is on top of the body (701), while the head 704 is to theside of the body when falling sideways (702). The head 704 is on thebottom when falling towards the camera (703). The speed and duration ofthe change in head location and position may help to avoid false alarms:if a person bends down, as in FIG. 4, the head location and relativeposition changes similarly to a falling event (702). Falling, however,is typically more sudden. The speed and duration of the change in theposition of the head must meet predefined limits for a falling event tobe indicated. The limits depend on the specific implementation.

Detailed human body modeling and tracking may provide further data forthe detection of falling and getting up events. Using methods known fromthe art may provide a detailed description of the body, including thelocation and position of not only the head but also that of other majorbody parts. Major body parts may be defined to include the head, torso,the individual limbs. The IVS system may learn the relative position andmotion of these major body parts, and possibly the position and motionof other body parts as well, during falling and getting up from trainingsequences and use this information to detect events.

The methods described above can be implemented using the system depictedin FIG. 8. FIG. 8 illustrates a plan view of an exemplary videosurveillance system. A computer system 801 comprises a computer 802having a computer-readable medium 803 embodying software to operate thecomputer 802 according to the invention. The computer system 801 iscoupled to one or more video sensors 804, one or more video recorders805, and one or more input/output (I/O) devices 806. The video sensors804 can also be optionally coupled to the video recorders 805 for directrecording of video surveillance data. The computer system is optionallycoupled to other sensors 807.

The video sensors 804 provide source video to the computer system 801.Each video sensor 804 can be coupled to the computer system 801 using,for example, a direct connection (e.g., a firewire digital camerainterface) or a network. The video sensors 804 can exist prior toinstallation of the invention or can be installed as part of theinvention. Examples of a video sensor 804 include: a video camera; adigital video camera; a color camera; a monochrome camera; a camera; acamcorder, a PC camera; a webcam; an infra-red video camera; and a CCTVcamera.

The video recorders 805, which are optional, receive video surveillancedata from the computer system 801 for recording and/or provide sourcevideo to the computer system 801. Each video recorder 805 can be coupledto the computer system 801 using, for example, a direct connection or anetwork. The video recorders 805 can exist prior to installation of theinvention or can be installed as part of the invention. The videosurveillance system in the computer system 11 may control when and withwhat quality setting a video recorder 805 records video. Examples of avideo recorder 805 include: a video tape recorder; a digital videorecorder; a network video recorder; a video disk; a DVD; and acomputer-readable medium.

The I/O devices 806 provide input to and receive output from thecomputer system 801. The I/O devices 806 can be used to task thecomputer system 801 and produce reports from the computer system 801.Examples of I/O devices 806 include: a keyboard; a mouse; a stylus; amonitor; a printer; another computer system; a network; and an alarm.For example, notification of a falling event may be provided toemergency response personnel, such as fire and rescue personnel, storesecurity personnel, and others by sending a notification to therespective computer systems or by sounding an alarm.

The embodiments and examples discussed herein are non-limiting examples.

The invention is described in detail with respect to preferredembodiments, and it will now be apparent from the foregoing to thoseskilled in the art that changes and modifications may be made withoutdeparting from the invention in its broader aspects, and the invention,therefore, as defined in the claims is intended to cover all suchchanges and modifications as fall within the true spirit of theinvention.

1. A method, comprising: receiving input video data; processing thevideo data to detect a change in a posture of a human target in theinput video data.
 2. The method of claim 1, wherein the change in theposture is detected based on at least one of a height or an aspect ratioof the human target.
 3. The method of claim 2, wherein the change in theposture is detected based on at least a location of a top and a bottomof the human target.
 4. The method of claim 2, wherein the change in theposture is detected based on at least a trajectory of the human target.5. The method of claim 1, further comprising tracking a head of thehuman target.
 6. The method of claim 5, wherein the change in theposture is detected based on at least a location of the head.
 7. Amethod, comprising: receiving input video data; detecting a target inthe video data; tracking the target; identifying a target as a human;and detecting a change in a posture of the human target in the videodata.
 8. The method of claim 7, wherein the detecting the change stepcomprises detecting the change in posture when a change in at least oneof a height or aspect ratio of the human target exceeds a predeterminedthreshold.
 9. The method of claim 8, wherein the detecting stepcomprises: determining the height or the aspect ratio of the target in afirst frame of the video data; determining the height or aspect ratio ofthe target in a second frame of the video data; calculating a differencebetween the height or aspect ratio of the target in the first and secondframes; and detecting the change in the posture when either differenceexceeds a predetermined threshold.
 10. The method of claim 9, furthercomprising: determining an elapsed time between the first and secondframes; and detecting the change in the posture when the elapsed time isbelow a predetermined threshold.
 11. The method of claim 10, furthercomprising: determining a second change in the posture has occurredbased on a height or aspect ratio of the human target; determining alength of time between the change in posture and the second change inposture; and detecting the change in the posture when the elapsed timeis below a predetermined threshold.
 12. The method of claim 9, furthercomprising: determining the height and aspect ratio of the target in afirst frame and a third frame of the video data; determining the lengthof time between the first and third frames; and detecting the change inposture when the time is greater than a predetermined threshold.
 13. Themethod of claim 9, further comprising: tracking a top and bottom of thetarget; and detecting the change in the posture based at least in parton the location of the top and bottom of the target.
 14. The method ofclaim 9, wherein the change in the posture is detected when the top ofthe target moves but the bottom remains in substantially the samelocation.
 15. The method of claim 8, further comprising: measuring aheight of the target; and detecting the change in posture based on achange in the height of the target.
 16. The method of claim 8, furthercomprising tracking a head of the human target.
 17. The method of claim16, wherein the change in the posture is detected based on at least alocation of the head.
 18. The method of claim 8, further comprising:determining a location and position of body parts of the human target;and detecting the change in posture based on the location and positionof the body parts.
 19. The method of claim 18, further comprising:tracking the position and movement of major body parts during a trainingsequence; and detecting the change in posture based at least in part ona comparison of the position and movement of the major body parts in thetraining sequence.
 20. A method, comprising: receiving input video dataof a place including a plurality of frames; tracking objects in theframes; classifying objects in the frames as human and non-humantargets; identifying at least one of a position or location of a humantarget in the frames; determining changes in the location and positionof the human target between frames; and detecting a falling event or agetting up event when the change in at least one of the position orlocation exceeds a predetermined threshold.
 21. The method of claim 20,wherein the identifying step comprises measuring at least one of aheight or aspect ratio of the human target, wherein the detecting stepcomprises calculating a difference in at least one of the height oraspect ratio between frames; and determining the change in position orlocation based on the difference in at least one of the aspect ratio andheight.
 22. The method of claim 21, further comprising: determining atrajectory of the human target; predicting at least one of a height andposition of the human target at a point in time; comparing at least oneof the predicted height or position to the actual height or position;and detecting the falling event or the getting up event based on atleast a difference between the predicted and actual height or position.23. The method of claim 22, further comprising utilizing calibrationinformation to predict the height or position of the human target. 24.The method of claim 20, wherein the determining step comprises trackinga top and bottom of the target; and detecting the falling event orgetting up event based at least in part on movement of the top andbottom of the target.
 25. The method of claim 24, wherein the fallingevent is detected when the top of the target moves but the bottomremains in substantially the same location.
 26. The method of claim 20,wherein the determining step comprises: determining a height of thetarget; and detecting the falling event or getting up event based atleast in part on a change in the height of the target.
 27. The method ofclaim 20, further comprising tracking a head of the human target. 28.The method of claim 27, further comprising detecting the falling eventor getting up event based at least in part on a location of the head.29. The method of claim 20, further comprising: measuring at least oneof a speed or duration of the change is aspect ratio or height; anddetecting the falling event or getting up event based at least in parton the speed or duration of the change.
 30. The method of claim 20,wherein the place is one of a hospital, store, parking lot, mall, orassisted living community.
 31. The method of claim 20, furthercomprising sending an alert regarding the occurrence of the falling downor getting up event.
 32. The method of claim 31, wherein the alert isprovided to at least one of hospital staff, security personnel,emergency response personnel, law enforcement.
 33. A computer-readablemedium containing instructions that when executed by a computer systemcause said computer system to implement the method according to claim.34. A video processing system comprising: the computer-readable mediumaccording to claim 33; and a computer coupled to said computer-readablemedium to execute the instructions contained on said computer-readablemedium.
 35. A method of detecting a person falling or getting up,comprising: receiving video data of a place from a video camera;processing the video data to track a persons movements in the place;determining if the person falls or gets up based on the processing ofthe video data; sending an signal indicating the person has fallen orgotten up to an output device when it is determined that the personfalls or gets up;
 36. The method of claim 35, wherein the place is oneof a mall, store, parking lot, assisted living community, hospital,home, or law enforcement environment.